Method and system for automatically tracking and managing inventory of surgical tools in operating rooms

ABSTRACT

Embodiments described herein provide various examples of automatically processing surgical videos to detect surgical tools and tool-related events, and extract surgical-tool usage information. In one aspect, a process for automatically tracking usages of robotic surgery tools is disclosed. This process can begin by receiving a surgical video captured during a robotic surgery. The process then processes the surgical video to detect a surgical tool in the surgical video. Next, the process determines whether the detected surgical tool has been engaged in the robotic surgery. If so, the process further determines whether the detected surgical tool is engaged for a first time in the robotic surgery. If the detected surgical tool is engaged for the first time, the process subsequently increments a total-engagement count of the detected surgical tool. Otherwise, the process continues monitoring the detected surgical tool in the surgical video.

PRIORITY CLAIM AND RELATED PATENT APPLICATIONS

This patent application is a continuation of, and hereby claims thebenefit of priority under 35 U.S.C. § 120 to co-pending U.S. patentapplication Ser. No. 16/129,607, filed on 12 Sep. 2018, entitled,“Method and Apparatus for Automatically Tracking and Managing Inventoryof Surgical Tools in Operating Rooms,” by inventor JagadishVenkataraman. The above-listed application is hereby incorporated byreference as a part of this patent document.

TECHNICAL FIELD

The present disclosure generally relates to building surgical videoanalysis tools and, more specifically, to systems, devices andtechniques for automatically segmenting surgical videos, detectingsurgical tools and tool-related events, and extracting surgical-toolusage information to guide or make recommendations to the surgeons.

BACKGROUND

Tracking the use and managing the inventory of surgical tools in theoperating rooms (ORs) play a highly important role in improving surgicaloutcomes and delivering quality patient care. For example, certainsurgical tools have limited shelf life in terms of the number of timesthey can be fired. Hence, it is necessary to track and record the usagesof these surgical tools. Currently, tracking and recording the usages ofsurgical tools in the ORs involve the surgical staff manually performingthese tasks using either a computer or by making a note on thewhiteboard. As another example, for minimally invasive procedures (MIS)such as a laparoscopy surgery, it is useful to detect which tools areengaged for tool inventory purposes. Today, information regarding whichtools are engaged during a surgical procedure is also collected manuallyby the surgical staff. In fact, most of the surgical tool-relatedmetrics in the ORs are collected manually by the surgical staff.Unfortunately, manually counting or tracking surgical-tool-relatedmetrics for each and every surgical procedure is highly labor-intensiveand error-prone.

SUMMARY

Embodiments described herein provide various examples of asurgical-tool-management system for automatically processing surgicalvideos to detect surgical tools and tool-related events, and extractsurgical-tool usage information to guide or make recommendations tosurgical staff. In one aspect, a process for automatically trackingusages of robotic surgery tools is disclosed. This process can begin byreceiving a surgical video captured during a robotic surgery. Theprocess then processes the surgical video to detect a surgical tool inthe surgical video. Next, the process determines whether the detectedsurgical tool has been engaged in the robotic surgery. If so, theprocess further determines whether the detected surgical tool is engagedfor a first time in the robotic surgery. If the detected surgical toolis engaged for the first time, the process subsequently increments atotal-engagement count of the detected surgical tool. Otherwise, theprocess continues monitoring the detected surgical tool in the surgicalvideo.

In some embodiments, processing the surgical video to detect thesurgical tool includes applying a machine-learning model on a set ofvideo images in the surgical video, wherein the machine-learning modelis trained to detect and identify a set of surgical tools associatedwith the robotic surgery.

In some embodiments, the process determines whether the detectedsurgical tool has been engaged in the robotic surgery by firstdetermining if the detected surgical tool continues to present in thesurgical video for at least a predetermined minimum time period. If so,the process determines that the surgical tool has been engaged in therobotic surgery. Otherwise, the process continues monitoring thesurgical tool in the surgical video.

In some embodiments, prior to determining whether the detected surgicaltool has been engaged in the robotic surgery, the process determines,for each type of a set of surgical tool types, a minimum time periodrequired to engage the surgical tool type in a surgical procedure.

In some embodiments, if the detected surgical tool has been previouslyengaged in the robotic surgery, the process continues monitoring othersurgical tools in the robotic surgery.

In some embodiments, after incrementing the total-engagement count ofthe detected surgical tool, the process records the updated engagementcount of the detected surgical tool for tool inventory purposes.

In some embodiments, if the updated engagement count has reached amaximum number of engagements for the safe and/or effective use of thedetected surgical tool, the process marks the detected surgical tool asexpired.

In some embodiments, after determining that the detected surgical toolhas been engaged in the robotic surgery, the process activates atool-usage counting process to track and record a number of uses of thedetected surgical tool during the tool engagement.

In some embodiments, the tool-usage counting process tracks the numberof uses of the detected surgical tool by using a second machine-learningmodel trained to detect an event in the surgical video representing asingle use of the detected surgical tool.

In some embodiments, the detected surgical tool is a surgical stapler,and the second machine-learning model is trained to detect a singlefiring of the surgical stapler each time a firing knob of the surgicalstapler is pushed from a first position to a second position.

In some embodiments, the detected surgical tool is an electrocauterytool, and the second machine-learning model is trained to detect asingle firing of the electrocautery tool each time a plume of surgicalsmoke is detected.

In some embodiments, the process can further receive an existing numberof total uses of the detected surgical tool from previous engagements ofthe detected surgical tool. The process subsequently updates the totaluses of the detected surgical tool by combining the recorded number ofuses during the tool engagement with the existing number of total usesfrom the previous engagements.

In some embodiments, the process further includes the steps of:determining if a maximum number of total uses of the detected surgicaltool has been reached; and if so, generating a tool-expirationnotification to be communicated to surgical staff.

In another aspect, an apparatus for automatically tracking usages ofrobotic surgery tools is disclosed. This apparatus includes one or moreprocessors and a memory coupled to the one or more processors. Thememory stores instructions that, when executed by the one or moreprocessors, cause the apparatus to: receive a surgical video capturedduring a robotic surgery; process the surgical video to detect asurgical tool in the surgical video; determine whether the detectedsurgical tool has been engaged in the robotic surgery; and if so,further determine if the detected surgical tool has beenpreviously-engaged in the robotic surgery. If the detected surgical toolhas not been previously-engaged in the robotic surgery, the memorystores instructions that, when executed by the one or more processors,cause the apparatus to increment a total-engagement count of thedetected surgical tool. Otherwise, the memory stores instructions that,when executed by the one or more processors, cause the apparatus tocontinue monitoring the detected surgical tool in surgical video.

In some embodiments, the memory further stores instructions that, whenexecuted by the one or more processors, cause the apparatus to activatea tool-usage counting process to track and record a number of uses ofthe detected surgical tool during the tool engagement after determiningthat the detected surgical tool has been engaged.

In some embodiments, the memory further stores instructions that, whenexecuted by the one or more processors, cause the apparatus to use asecond machine-learning model to track the number of uses of thedetected surgical tool, wherein the second machine-learning model istrained to detect an event in the surgical video representing a singleuse of the detected surgical tool.

In some embodiments, the memory further stores instructions that, whenexecuted by the one or more processors, cause the apparatus to: receivean existing number of total uses of the detected surgical tool fromprevious engagements of the detected surgical tool; and update the totaluses of the detected surgical tool by combining the recorded number ofuses during the tool engagement with the existing number of total usesfrom the previous engagements.

In some embodiments, the memory further stores instructions that, whenexecuted by the one or more processors, cause the apparatus to:determine if a maximum number of total uses of the detected surgicaltool has been reached; and if so, generate a tool-expirationnotification to be communicated to surgical staff.

In yet another aspect, a robotic surgical system is disclosed. Thisrobotic surgical system includes: one or more surgical tools eachcoupled to a respective robotic arm; an endoscope configured to capturesurgical videos; and a controller coupled to each robotic arm and theendoscope. The controller is configured to: receive a surgical videocaptured by the endoscope during a robotic surgery; process the surgicalvideo to detect a surgical tool in the surgical video; determine whetherthe detected surgical tool has been engaged in the robotic surgery; andif so, further determine whether the detected surgical tool is engagedfor a first time in the robotic surgery: if so, increment atotal-engagement count of the detected surgical tool; otherwise,continue monitoring the detected surgical tool in the surgical video.

In some embodiments, the controller is further configured to: detectthat the total-engagement count has reached a maximum number ofengagements for the safe and/or effective use of the detected surgicaltool and subsequently mark the detected surgical tool as expired andupdate a status of the detected surgical tool for tool inventorypurposes.

BRIEF DESCRIPTION OF THE DRAWINGS

The structure and operation of the present disclosure will be understoodfrom a review of the following detailed description and the accompanyingdrawings in which like reference numerals refer to like parts and inwhich:

FIG. 1 shows a block diagram of an exemplary surgical-tool-managementsystem 100 in accordance with some embodiments described herein.

FIG. 2 presents a flowchart illustrating an exemplary process forprocessing a surgical video to detect a new surgical tool engagementduring a recorded surgical procedure in accordance with some embodimentsdescribed herein.

FIG. 3 presents a flowchart illustrating an exemplary process forprocessing a surgical video to track a number of uses of a limited-usesurgical tool in accordance with some embodiments described herein.

FIG. 4 shows a photographic image depicting a surgical stapler thatincludes a ruler printed on the cartridge section of the surgicalstapler in accordance with some embodiments described herein.

FIG. 5 conceptually illustrates a computer system with which someembodiments of the subject technology can be implemented.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofvarious configurations of the subject technology and is not intended torepresent the only configurations in which the subject technology may bepracticed. The appended drawings are incorporated herein and constitutea part of the detailed description. The detailed description includesspecific details for the purpose of providing a thorough understandingof the subject technology. However, the subject technology is notlimited to the specific details set forth herein and may be practicedwithout these specific details. In some instances, structures andcomponents are shown in block diagram form in order to avoid obscuringthe concepts of the subject technology.

Recorded videos of medical procedures such as surgeries contain highlyvaluable and rich information for medical education and training,assessing and analyzing the quality of the surgeries and skills of thesurgeons, and for improving the outcomes of the surgeries and skills ofthe surgeons. There are many surgical procedures that involve displayingand capturing video images of the surgical procedures. For example,almost all minimally invasive procedures (MIS), such as endoscopy,laparoscopy, and arthroscopy, involve using video cameras and videoimages to assist the surgeons. Furthermore, the state-of-the-artrobotic-assisted surgeries require intraoperative video images beingcaptured and displayed on the monitors for the surgeons. Consequently,for many of the aforementioned surgical procedures, e.g., a gastricsleeve or cholecystectomy, a large cache of surgical videos alreadyexists and continues to be created as a result of a large number ofsurgical cases performed by many different surgeons from differenthospitals. The simple fact of the existence of a huge (and constantlyincreasing) number of surgical videos of a particular surgical proceduremakes processing and analyzing the surgical videos of the givenprocedure a potential machine-learning problem.

This patent disclosure provides various examples of applyingmachine-learning-based and/or computer-vision-based techniques toautomatically perform various surgical tool-related tasks, such as tooldetection, tool usage counting and tracking, and outcome analysis, andto automatically generate alerts/guidance to the surgical staff. Thedisclosed automated techniques relieve surgical staff from manuallyperforming the above tasks, thereby freeing up the valuable time ofsurgical staff for other more important tasks, easing the surgicalworkflow, and improving surgical outcomes by eliminating manual errors.

In various embodiments, the tool-related data can be mined from alltypes of surgical video feeds, including but not limited to open surgeryvideos, endoscopic surgery videos, and robotic surgery videos. For eachtype of tool-related data, data can be mined from such videos usingvarious machine-learning-based techniques, conventional computer-visiontechniques such as image processing, or a combination ofmachine-learning and Computer-vision techniques.

FIG. 1 shows a block diagram of an exemplary surgical-tool-managementsystem 100 in accordance with some embodiments described herein. As canbe seen in FIG. 1 , surgical-tool-management system 100 includes atleast a video source 102 and a video-analysis subsystem 104. In someembodiments, video source 102 can include live surgical procedurevideos. In these embodiments, video source 102 can be coupled to or partof a live video capturing module configured to capture and recordreal-time surgical procedure videos and/or still images during livesurgical procedures. Moreover, this live video capturing module can becoupled to or part of a surgical platform, such as an open surgeryplatform, an endoscopic surgery platform, or a robotic surgery platform.As such, video source 102 can generate and provide various types ofsurgical videos, including but not limited to open surgery videos,endoscopic surgery videos, and robotic surgery videos. In someembodiments, video source 102 can include stored/archived surgicalprocedure videos. In these embodiments, video source 102 can be coupledto or part of a storage system or device that stores previously capturedsurgical procedure videos and/or still images. As such, video source 102can store and provide various types of surgical videos, including butnot limited to open surgery videos, endoscopic surgery videos, androbotic surgery videos.

As can be seen in FIG. 1 , video-analysis subsystem 104 is coupled tovideo source 102 and configured to receive live or stored surgicalvideos and/or still images (collectively referred to as “surgical videoimages 130” hereinafter) from video source 102 and perform varioussurgical video analyses to extract tool-related information. Morespecifically, video-analysis subsystem 104 includes a set ofvideo-processing modules 108, wherein each of the set ofvideo-processing modules 108 can be used to process surgical videoimages 130 to extract one or more particular types of tool-relatedinformation.

For example, video-processing modules 108 can include a tool-engagementdetection module 108-1 for processing surgical video images 130 anddetecting which surgical tools have been engaged during a recordedsurgical procedure. In various embodiments, tool-engagement detectionmodule 108-1 can include a machine-learning or a computer-visiontechnique trained to detect and identify various surgical tools.Tool-engagement detection module 108-1 can employ this machine-learningor computer-vision technique to detect various surgical tools capturedin surgical video images 130. Hence, tool-engagement detection module108-1 operates to automatically identify different types of toolengagements captured in surgical video images 130 and automaticallygenerate an inventory of the detected tools involved in the associatedsurgical procedure.

Video-processing modules 108 can include a tool-usage counting module108-2 for processing surgical video images 130 to determine how manytimes a particular surgical tool has been fired or used in anothermanner. Note that certain surgical tools, such as surgical staplers orsome power-based surgical tools (e.g., electrocautery tools), havelimited uses in terms of the number of times they can be fired orengaged/used in other manners. For example, a given surgical stapler ora given electrocautery tool can only be fired a limited number of timesbefore the tool has to be replaced. As a result, it is highly desirableto keep track of the number of firings/uses/applications of a givensurgical tool having limited use Such a surgical tool having limited useis also referred to as a “limited-use surgical tool” hereinafter.Conventionally, this usage data is tracked manually by surgical stafffor certain surgical procedures, such as endoscopic procedures. Invarious embodiments, tool-usage counting module 108-2 can include amachine-learning or a computer-vision technique trained to identify andkeep track of each time a given surgical tool is fired. Tool-usagecounting module 108-2 can employ this machine-learning orcomputer-vision technique to automatically identify each time a givenlimited-use surgical tool is fired or used in another manner, therebykeeping track of the usage of the given surgical tool. For example, fora surgical stapler, a machine-learning model can be trained to detecteach time a firing knob of the surgical stapler is pushed from a firstposition to a second position to detect a single firing of the surgicalstapler. As another example, for a powered cautery tool, amachine-learning model can be trained to detect each time a plume ofsurgical smoke is generated as a single firing of the cautery tool. Notethat this usage information is accumulative and tool-usage countingmodule 108-2 can combine the usage information extracted from surgicalvideo images 130 from multiple surgical videos.

In some embodiments, when a maximum usage (e.g., the maximum number offirings of a surgical stapler) is determined to have been reached,tool-usage counting module 108-2 can generate an alert or a notificationto be communicated to surgical staff (described in more detail below).In some embodiments, tool-usage counting module 108-2 can also be usedto count events such as the number of staple cartridges used or thenumber of sponges used. Hence, tool-usage counting module 108-2 operatesto automatically determine how many times a particular surgical tool hasbeen used or the number of time a given type of surgical item has beenapplied without requiring human intervention or manual processing toeliminate human errors.

In some embodiments, tool-usage counting module 108-2 can be configuredto infer a length of the tissue traversed by a surgical tool. Forexample, based on the determined number of uses of a surgical tool, suchas a surgical stapler, and a length of the surgical tool, tool-usagecounting module 108-2 can quickly estimate the approximate length of atissue that the surgical tool has traversed along the tissue. Bycombining the automatic tool-usage counting and the length estimate, theapproximate length of the tissue that the surgical tool has traversedcan be quickly estimated without the need to manually measure thetraversed length.

Video-processing modules 108 can also include a tool-parameter detectionmodule 108-3 for processing surgical video images 130 to determine agiven parameter associated with a given surgical tool. For example, thecolor of the surgical stapler cartridge/load is an important parameterto collect because different colors of the cartridge/load representdifferent types of staples for different tissue thicknesses. The colorof the cartridge/load is also an important parameter during apost-operation summary procedure to summarize what types of staplercartridge/load have been used on the patient during the procedure.Conventionally, the color of the cartridge/load is tracked visually andmanually by surgical staff. Some future surgical stapler designs aim totransmit the color information as a feedback signal through the surgicalstapler.

In various embodiments, tool-parameter detection module 108-3 caninclude a machine-learning or a computer-vision model trained toidentify the color of each surgical stapler load. Tool-parameterdetection module 108-3 can employ this machine-learning orcomputer-vision model to automatically identify the color of eachsurgical stapler load captured in surgical video images 130, therebygreatly simplifying the color identification/tracking process withoutthe need to make a design change for the surgical stapler tool just tocollect the color information.

Video-processing modules 108 can also include a staple-line verificationmodule 108-4 for processing surgical video images 130 to verifystability and uniformity of each staple line formed in a tissue or abody part. Note that malformed staple lines can result in leakage,especially when multiple malformed staples group or align in a certaingeometrical pattern. Staple-line verification module 108-4 can include amachine-learning or a computer-vision model trained to identifymalformed staples and compute their relative geometry. Staple-lineverification module 108-4 can employ this machine-learning orcomputer-vision model to automatically process surgical video images 130including the formed staple lines and to identify malformed staples anddetermine their relative geometry. This determined geometry of malformedstaples can then be used to predict a probability of leakage.

Video-processing modules 108 can also include a loose-object detectionmodule 108-5 for processing surgical video images 130 to identity looseor floating objects such as loose or open staples in the tissue. Duringthe operation of a surgical stapler, the knife in the stapler that cutsthrough a tissue between two rows of formed staple lines can sometimesaccidentally cut through one of the already formed staple linesresulting in “crotch staples,” i.e., loose, V-shaped staples that canlater float around in the patient's body. If not removed, these loosestaples have the ability to move around, puncture other tissues andcause bleeding. In various embodiments, loose-object detection module108-5 can include a machine-learning or a computer-vision model trainedto identify “crotch staples” or other forms of loose staples.Loose-object detection module 108-5 can employ this machine-learning orcomputer-vision model to automatically process surgical video images 130of the formed staple lines and to identify “crotch staples” and otherforms of loose staples. This determined geometry of malformed staplescan then be used to predict potential leaks. In some embodiments, whenloose staples are detected, loose-object detection module 108-5 cangenerate an alert or a notification to be communicated to surgical staff(described in more detail below). The surgical staff can then takeproper action to remove these loose staples, thus avoiding potentialinjuries.

Note that various video-processing modules 108-1 to 108-5 are just someexamples of surgical-video-processing modules for automaticsurgical-tool management. It can be understood that, over time,video-processing modules 108 can include an increasingly large number ofmodules to automatically process video images and extract even moretypes of surgical-tool-related information.

As illustrated in FIG. 1 , video-processing modules 108 can generatealert signals 110 to surgical staff 140 in real time. Surgical staff 140can then take proper action based on the nature of alert signals 110.For example, when a maximum usage of a surgical tool is detected,tool-usage counting module 108-2 can generate an alert signal 110 toalert surgical staff 140. Surgical staff 140 can then replace the oldsurgical tool with a new surgical tool. In some embodiments, alertsignals 110 can also directly trigger the surgical tool to be lockedfrom further operation.

Detection of Tool Engagement with the Tool-Engagement Detection Module

Conventionally, tracking surgical tool engagements during a surgicalprocedure requires the surgical staff to visually track each new toolengagement and log information into a PC or make notes on thewhiteboard. Using machine-learning, computer-vision, or a combinedtechnique of machine-learning and computer-vision, a computer can beprogrammed or trained to detect the occurrences and subsequentlyidentify the tool type for different tools in the surgical videos. Theproposed tool-engagement detection module 108-1 can process surgicalvideos and detect which surgical tools have been engaged during arecorded surgical procedure.

In some embodiments, the video processing outputs from the disclosedtool-engagement detection module can be corroborated with feedback datacollected directly from the surgical systems. For example, in somerobotic surgery systems, each time a surgical tool is mounted on arobotic arm, the tool information, such as tool type and tool ID, can beobtained through an interface of the robotic arm, and this toolinformation can be logged and stored. In some embodiments, toolinformation collected by the surgical robot can be combined with thetool detection output of the disclosed tool-engagement detection moduleto provide a more complete record for each detected tool engagement, andfor more accurate tool tracking.

FIG. 2 presents a flowchart illustrating an exemplary process 200 forprocessing a surgical video to detect a new surgical tool engagementduring a recorded surgical procedure in accordance with some embodimentsdescribed herein. In one or more embodiments, one or more of the stepsin FIG. 2 may be omitted, repeated, and/or performed in a differentorder. Accordingly, the specific arrangement of steps shown in FIG. 2should not be construed as limiting the scope of the technique. Notethat process 200 can be implemented within the above-describedtool-engagement detection module 108-1.

Process 200 begins by receiving a surgical video captured for thesurgical procedure (step 202). Note that the surgical video can be anopen surgery video, an endoscopic surgery video, or a robotic surgeryvideo. Next, process 200 segments the surgical video into a set of videosequences, wherein each of the video sequences is composed of a sequenceof video frames (step 204). For example, process 200 can first segmentthe surgical video into a set of surgical phases, and further segment agiven surgical phase in the set of surgical phases into a set ofsurgical subphases or surgical tasks. More detail of segmenting asurgical video into phases, subphases and/or tasks is described in arelated patent application entitled “Machine-Learning-Oriented SurgicalVideo Analysis System,” having Ser. No. 15/987,782, and filing date of23 May 2018, the content of which is incorporated herein by reference.In some embodiments, each of the segmented surgical phases, subphases,and tasks can be further segmented into a set of equal-length videosequences, wherein each of the sequences includes a predetermined numberof video frames.

For each sequence of video frames in the set of video sequences, process200 then processes the sequence of video frames to detect one or moresurgical tool engagements associated with one or more detected surgicaltools in the sequence of video frames (step 206). In some embodiments,to detect a tool engagement, process 200 first detects a surgical toolin the sequence of video frames using a machine-learning model trainedto detect and identify a set of surgical tools. For example, themachine-learning model can be trained to detect and identify each of aset of required surgical tools for a given surgical procedure capturedby the surgical video. In various embodiments, this machine-learningmodel can include a regression model, a deep neural network-based model,a support vector machine, a decision tree, a Naive Bayes classifier, aBayesian network, or a k-nearest neighbors (KNN) model. In someembodiments, this machine-learning model is constructed based on aconvolutional neural network (CNN) architecture, an recurrent neuralnetwork (RNN) architecture, or another form of deep neural network (DNN)architecture.

Note that processing multiple consecutive video frames to detect asurgical tool engagement can be more accurate than processing a singlevideo frame. This is because a meaningful engagement of a surgical toolis often associated with a minimum time period. Hence, when a surgicaltool appears in the video for a time period shorter than the minimumtime period, the surgical tool may not be actually engaged but simplypresented for a different reason in the video. For example, if asurgical stapler appears in a sequence of video frames for less than asecond then disappears, the surgical stapler is usually not engagedduring the short appearance. Note that this minimum time period can bemeasured by a predetermined number of consecutive video frames. Hence,in some embodiments, after a surgical tool is detected in the sequenceof video frames being processed, the detected surgical tool isdetermined to be a tool engagement only when the surgical tool remainsbeing detected for the predetermined number of consecutive video frames.Those skilled in the art will appreciate that this minimum time periodcan vary significantly for different types of surgical tools. Therefore,after a given surgical tool has been detected, a corresponding minimumtime period for the given surgical tool will be applied to determine ifthe given surgical tool is actually engaged. Note that because therecorded number of engagements of a given surgical tool is directlyrelated to calculating the remaining shelf life/number-of-uses of thegiven surgical tool, it is necessary to distinguish actual toolengagements from accidental appearances of the given surgical tool.

At step 208, process 200 determines if a surgical tool engagement hasbeen detected. If not, process 200 terminates or returns to step 206 toprocess the next sequence of video frames in the set of video sequences.If a surgical tool engagement is detected in the current sequence ofvideo frames, process 200 then determines if the detected surgical toolassociated with the detected tool engagement is associated with apreviously identified surgical tool engagement in a previously-processedsequence of video frames in the set of video sequences (step 210). Ifso, process 200 terminates or returns to step 206 to process the nextsequence of video frames in the set of video sequences. However, if thedetected surgical tool engagement has not been detected in thepreviously-processed sequences of video frames, process 200 determinesthat a new surgical tool engagement has been detected and subsequentlyrecords the detected tool engagement for tool inventory purposes (step212).

Note that the above-described process of taking tool inventory regardingwhich surgical tools have been engaged during a surgical procedure canbe combined with a tool-usage counting process to track the number oftimes a given surgical tool has been used. More specifically, for agiven surgical tool with limited use, each time the surgical tool isdetected and associated with a newly-identified surgical toolengagement, the tool-usage counting process can be activated to trackand record the number of individual uses/applications of the givensurgical tool to tissues during the newly-identified tool engagement.Note that for the given surgical tool with limited use, the usagecounting is accumulative over each newly-identified engagement andpreviously-identified engagements of the given surgical tool, if any. Insome embodiments, when the accumulative number of uses of the givensurgical tool has reached a predetermined number of uses (e.g., themaximum number of effective uses or the maximum number of safe uses),the given surgical tool is considered expired and a replacement warningfor the given surgical tool can be issued to the surgical staff. Anexemplary process of tracking a total number of times a given surgicaltool has been used is described below in conjunction with FIG. 3 .

Counting Tool Usage with the Tool-Usage Counting Module

As mentioned above, certain surgical tools, such as a surgical stapler,have limited uses in terms of the number of times they can be fired orused in other manners. Conventionally, this usage data is trackedvisually and manually by surgical staff for certain surgical procedures,such as endoscopic procedures. Using machine-learning, computer-vision,or a combined technique of machine-learning and computer-vision, acomputer can be programmed or trained to identify and keep track of eachtime a target surgical tool is fired or used in another manner. Thedisclosed tool-usage counting module 108-2 can use model-basedtechniques to process surgical videos and identify and keep track ofeach time a target surgical tool is fired or used in another manner ineach new application.

In some embodiments, the video processing outputs from the disclosedtool-usage counting module can be corroborated with feedback datacollected directly from the surgical systems. For example, in somerobotic surgery systems, each time a surgical tool is mounted on arobotic arm, the tool information, such as tool type and tool ID can beobtained through an interface of the robotic arm, and this toolinformation can be logged and stored. In these embodiments, toolinformation collected by the surgical robots can be processed todetermine the number of times a target surgical tool has been mounted onone or more surgical robots. This data can then be combined with thetool tracking output from the disclosed tool-usage counting module 108-2for the same surgical tool to provide a more accurate usage count forthe surgical tool.

FIG. 3 presents a flowchart illustrating an exemplary process 300 forprocessing a surgical video to track a number of uses of a limited-usesurgical tool in accordance with some embodiments described herein. Inone or more embodiments, one or more of the steps in FIG. 3 may beomitted, repeated, and/or performed in a different order. Accordingly,the specific arrangement of steps shown in FIG. 3 should not beconstrued as limiting the scope of the technique. Note that process 300can be implemented within the above-described tool-usage counting module108-2. Note that a limited-use surgical tool described herein caninclude a surgical stapler, a power-based cautery tool, or some otherpower-based surgical tools.

Process 300 begins by receiving a surgical video containing one or moreapplications of a limited-use surgical tool to tissues (step 302). Notethat the surgical video can be an open surgery video, an endoscopicsurgery video, or a robotic surgery video. Process 300 then processesthe video frames in the surgical video to detect that the limited-usesurgical tool is initially engaged (step 304). For example, a trainedmachine-learning model based on a set of images of the limited-usesurgical tool as training data can be used to detect and identify alimited-use surgical tool in step 304. In various embodiments, thismachine-learning model can include a regression model, a deep neuralnetwork-based model, a support vector machine, a decision tree, a NaiveBayes classifier, a Bayesian network, or a k-nearest neighbors (KNN)model. In some embodiments, this machine-learning model is constructedbased on a CNN architecture, an RNN architecture, or another form of DNNarchitecture. Note that during a surgical procedure captured in thesurgical video, there can exist multiple engagements of the samelimited-use surgical tool separated by other surgical tasks. Hence, step304 should be interpreted as detecting each new engagement of thelimited-use surgical tool rather than just the very first engagement ofthe limited-use surgical tool in the surgical video.

Next, during a time period when the detected limited-use surgical toolremains engaged, process 300 detects and tracks a number of uses of thelimited-use surgical tool (step 306). For example, after an engagementof the limited-use surgical tool was initially detected, amachine-learning model for detecting a single use of the limited-usesurgical tool can be activated. In some embodiments, for a surgicalstapler, this machine-learning model can be trained to detect each timea firing knob of the surgical stapler is pushed from a first position toa second position, which can be counted as a single use. As anotherexample, for a powered cautery tool, this machine-learning model can betrained to detect each time a plume of surgical smoke is generated as asingle use of the cautery tool. Note that, an operation which isconsidered as a single use for one type of limited-use surgical tool canbe significantly different from an operation which is considered as asingle use for another type of limited-use surgical tool.

After determining the number of uses of the limited-use surgical toolduring the current surgical tool engagement, the process next determinesif there is an existing count of uses of the limited-use surgical toolfrom the previously detected surgical tool engagements in the surgicalvideo (step 308). If so, process 300 combines the newly-counted numberof uses of the limited-use surgical tool during the current engagementwith the existing count of uses of the limited-use surgical tool (step310). In some embodiments, if the limited-use surgical tool operateswithin a robotic surgery system, then the limited-use surgical tool isinstalled onto a robot arm of the robotic surgery system when thelimited-use surgical tool is being used. In these embodiments, theexisting count of uses of the limited-use surgical tool can be stored ina storage device, such as a tool memory of the robotic surgery system.Hence, when the limited-use surgical tool is initially engaged with therobot arm, tool information such as tool ID, the existing count of usesof the limited-use surgical tool can be read off of the tool memory by atool controller of the robotic surgery system. Next, after the totalnumber of uses of the limited-use surgical tool has been updated withthe newly-counted number of uses, the updated total number of uses ofthe limited-use surgical tool can be written back into the tool memoryof the robotic surgery system by the tool controller to replace theexisting count of uses of the limited-use surgical tool.

After updating the total number of uses of the limited-use surgicaltool, process 300 next determines if a maximum number of uses of thelimited-use surgical tool has been reached (step 312). For example, themaximum number of uses of a surgical stapler can be determined based ona statistical number of effective uses of a surgical stapler before thesurgical stapler becomes unstable. In the robotic surgery systemexample, the maximum number of uses of the tool can also be stored inthe tool memory of the robotic surgery system and retrieved from thememory by the tool controller to be used by process 300 to execute step312. In some embodiments, once the maximum number of uses is reached,the limited-use surgical tool can be considered as expired, and process300 generates an expiration alert or notification to be communicated tothe surgical staff (step 314). Otherwise, process 300 terminates orreturns to step 304 to continue tracking the uses of the samelimited-use surgical tool (not shown).

As mentioned above, the counted number of uses/firings of a surgicalstapler during a given tool engagement can be used to infer a length ofthe tissue traversed by the surgical stapler. For example, theapproximate length of the tissue that the surgical stapler has traversedduring a single engagement can be estimated as the counted number offirings times the length of the cartridge section of the surgicalstapler. In some embodiments, the length of the cartridge can bedirectly extracted from an image of the surgical stapler, which shows aruler representing the length of the cartridge. For example, FIG. 4shows a photographic image depicting a surgical stapler 400, whichincludes a ruler 402 printed on the cartridge section of the surgicalstapler in accordance with some embodiments described herein. As shownin FIG. 4 , a ruler 402 indicates that for surgical stapler 400, eachcartridge is about 6 cm long. Hence, the disclosed tool-usage countingmodule 108-2 can also include a length estimation function to infer thelength of the tissue traversed by the surgical stapler based on thenumber of firings detected for a given engagement and the extractedlength information of the stapler cartridge.

Note that similarly to extracting the cartridge length information fromvideo images of a surgical stapler, other tool-related parameters canalso be extracted from video images for an identified surgical tool,such as a surgical stapler. For example, the color of the staplercartridge/load of a surgical stapler is an important parameter tocollect because different colors of the cartridge/load representdifferent types of staples for different tissue thicknesses. The colorof the cartridge/load is also an important parameter during apost-operation summary procedure to summarize what types of stapler loadhave been used on the patient during the surgical procedure. In someembodiments, the disclosed tool-usage counting module can include animage processing mechanism integrated with process 300 to determine thecolor of each stapler load for each detected firing of the surgicalstapler. The determined color information can be combined with othertool usage information, such as the length of the tissue traversed bythe stapler and the number of firings of that surgical stapler extractedfrom the surgical video. The combined tool usage information can then beused for post-operation analysis.

FIG. 5 conceptually illustrates a computer system with which someembodiments of the subject technology can be implemented. Computersystem 500 can be a client, a server, a computer, a smartphone, a PDA, alaptop, or a tablet computer with one or more processors embeddedtherein or coupled thereto, or any other sort of computing device. Sucha computer system includes various types of computer-readable media andinterfaces for various other types of computer-readable media. Computersystem 500 includes a bus 502, processing unit(s) 512, a system memory504, a read-only memory (ROM) 510, a permanent storage device 508, aninput device interface 514, an output device interface 506, and anetwork interface 516. In some embodiments, computer system 500 is apart of a robotic surgical system.

Bus 502 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices ofcomputer system 500. For instance, bus 502 communicatively connectsprocessing unit(s) 512 with ROM 510, system memory 504, and permanentstorage device 508.

From these various memory units, processing unit(s) 512 retrievesinstructions to execute and data to process in order to execute variousprocesses described in this patent disclosure, including theabove-described processes of detecting which surgical tools have beenengaged during a recorded surgical procedure, tracking how many times alimited-use surgical tool has been fired, and evaluating the quality ofa newly formed staple line in conjunction with FIGS. 1-3 . Theprocessing unit(s) 512 can include any type of processor, including, butnot limited to, a microprocessor, a graphics processing unit (GPU), atensor processing unit (TPU), an intelligent processor unit (IPU), adigital signal processor (DSP), a field-programmable gate array (FPGA),and an application-specific integrated circuit (ASIC). Processingunit(s) 512 can be a single processor or a multi-core processor indifferent implementations.

ROM 510 stores static data and instructions that are needed byprocessing unit(s) 512 and other modules of the computer system.Permanent storage device 508, on the other hand, is a read-and-writememory device. This device is a non-volatile memory unit that storesinstructions and data even when computer system 500 is off. Someimplementations of the subject disclosure use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) aspermanent storage device 508.

Other implementations use a removable storage device (such as a floppydisk, flash drive, and its corresponding disk drive) as permanentstorage device 508. Like permanent storage device 508, system memory 504is a read-and-write memory device. However, unlike storage device 508,system memory 504 is a volatile read-and-write memory, such as a randomaccess memory. System memory 504 stores some of the instructions anddata that the processor needs at runtime. In some implementations,various processes described in this patent disclosure, including theprocesses of detecting which surgical tools have been engaged during arecorded surgical procedure and tracking a number of uses of alimited-use surgical tool in conjunction with FIGS. 1-3 , are stored insystem memory 504, permanent storage device 508, and/or ROM 510. Fromthese various memory units, processing unit(s) 512 retrieve instructionsto execute and data to process in order to execute the processes of someimplementations.

Bus 502 also connects to input and output device interfaces 514 and 506.Input device interface 514 enables the user to communicate informationto and select commands for the computer system. Input devices used withinput device interface 514 include, for example, alphanumeric keyboardsand pointing devices (also called “cursor control devices”). Outputdevice interface 506 enables, for example, the display of imagesgenerated by computer system 500. Output devices used with output deviceinterface 506 include, for example, printers and display devices, suchas cathode ray tubes (CRT) or liquid crystal displays (LCD). Someimplementations include devices such as a touchscreen that functions asboth input and output devices.

Finally, as shown in FIG. 5 , bus 502 also couples computer system 500to a network (not shown) through a network interface 516. In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), an intranet, ora network of networks, such as the Internet. Any or all components ofcomputer system 500 can be used in conjunction with the subjectdisclosure.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedin this patent disclosure may be implemented as electronic hardware,computer software, or combinations of both. To clearly illustrate thisinterchangeability of hardware and software, various illustrativecomponents, blocks, modules, circuits, and steps have been describedabove generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present disclosure.

The hardware used to implement the various illustrative logics, logicalblocks, modules, and circuits described in connection with the aspectsdisclosed herein may be implemented or performed with a general-purposeprocessor, a digital signal processor (DSP), an application-specificintegrated circuit (ASIC), a field-programmable gate array (FPGA) orother programmable-logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor maybe a microprocessor, but, in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of receiver devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. Alternatively, some steps ormethods may be performed by circuitry that is specific to a givenfunction.

In one or more exemplary aspects, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored as one or moreinstructions or code on a non-transitory computer-readable storagemedium or non-transitory processor-readable storage medium. The steps ofa method or algorithm disclosed herein may be embodied inprocessor-executable instructions that may reside on a non-transitorycomputer-readable or processor-readable storage medium. Non-transitorycomputer-readable or processor-readable storage media may be any storagemedia that may be accessed by a computer or a processor. By way ofexample but not limitation, such non-transitory computer-readable orprocessor-readable storage media may include RAM, ROM, EEPROM, flashmemory, CD-ROM or other optical disk storage, magnetic disk storage orother magnetic storage devices, or any other medium that may be used tostore desired program code in the form of instructions or datastructures and that may be accessed by a computer. Disk and disc, asused herein, include compact disc (CD), laser disc, optical disc,digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disksusually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above are also includedwithin the scope of non-transitory computer-readable andprocessor-readable media. Additionally, the operations of a method oralgorithm may reside as one or any combination or set of codes and/orinstructions on a non-transitory processor-readable storage mediumand/or computer-readable storage medium, which may be incorporated intoa computer-program product.

While this patent document contains many specifics, these should not beconstrued as limitations on the scope of any disclosed technology or ofwhat may be claimed, but rather as descriptions of features that may bespecific to particular embodiments of particular techniques. Certainfeatures that are described in this patent document in the context ofseparate embodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in this patent document should not be understoodas requiring such separation in all embodiments.

Only a few implementations and examples are described, and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this patent document.

What is claimed is:
 1. A computer-implemented method for automaticallytracking usages of robotic surgery tools, the method comprising:receiving a surgical video captured during a robotic surgery; processingthe surgical video to detect a limited-use surgical tool in the surgicalvideo; determining whether the detected limited-use surgical tool hasbeen engaged in the robotic surgery, which includes determining if thedetected limited-use surgical tool continues to present in the surgicalvideo for at least a predetermined minimum time period by tracking thedetected limited-use surgical tool in the surgical video for apredetermined number of consecutive video frames corresponding to thepredetermined minimum time period; and if so, determining whether thedetected limited-use surgical tool has been previously engaged duringthe robotic surgery: if not, incrementing a total-engagement count ofthe detected limited-use surgical tool; otherwise, continuing monitoringthe detected limited-use surgical tool in the surgical video.
 2. Thecomputer-implemented method of claim 1, wherein processing the surgicalvideo to detect the limited-use surgical tool includes applying amachine-learning model on a set of video images in the surgical video,wherein the machine-learning model is trained to detect and identify aset of surgical tools associated with the robotic surgery.
 3. Thecomputer-implemented method of claim 1, wherein in response todetermining that the detected limited-use surgical tool has been engagedin the robotic surgery, the method further comprises continuingmonitoring the detected limited-use surgical tool in the surgical video.4. The computer-implemented method of claim 1, wherein prior todetermining whether the detected limited-use surgical tool has beenengaged in the robotic surgery, the method further comprises, for eachtype of a set of surgical tool types, determining a minimum time periodrequired to engage the surgical tool type in a surgical procedure. 5.The computer-implemented method of claim 1, wherein if the detectedlimited-use surgical tool has been previously engaged in the roboticsurgery, the method further includes continuing monitoring othersurgical tools in the robotic surgery.
 6. The computer-implementedmethod of claim 1, wherein after incrementing the total-engagement countof the detected limited-use surgical tool, the method further includesrecording the updated engagement count of the detected limited-usesurgical tool for tool inventory purposes.
 7. The computer-implementedmethod of claim 6, wherein if the updated engagement count has reached amaximum number of engagements for the safe and/or effective use of thedetected limited-use surgical tool, the method further comprises markingthe detected limited-use surgical tool as expired.
 8. Thecomputer-implemented method of claim 1, wherein after determining thatthe detected limited-use surgical tool has been engaged in the roboticsurgery, the method further comprises activating a tool-usage countingprocess to track and record a number of uses of the detected limited-usesurgical tool during the tool engagement.
 9. The computer-implementedmethod of claim 8, wherein using the tool-usage counting process totrack the number of uses of the detected limited-use surgical toolincludes using a second machine-learning model trained to detect anevent in the surgical video representing a single use of the detectedlimited-use surgical tool.
 10. The computer-implemented method of claim9, wherein the detected limited-use surgical tool is a surgical stapler,and wherein the second machine-learning model is trained to detect asingle firing of the surgical stapler each time a firing knob of thesurgical stapler is pushed from a first position to a second position.11. The computer-implemented method of claim 9, wherein the detectedlimited-use surgical tool is an electrocautery tool, and wherein thesecond machine-learning model is trained to detect a single firing ofthe electrocautery tool each time a plume of surgical smoke is detected.12. The computer-implemented method of claim 8, wherein the methodfurther comprises: receiving an existing number of total uses of thedetected limited-use surgical tool from previous engagements of thedetected limited-use surgical tool; and updating the total uses of thedetected limited-use surgical tool by combining the recorded number ofuses during the tool engagement with the existing number of total usesfrom the previous engagements.
 13. The computer-implemented method ofclaim 12, wherein the method further comprises: determining if a maximumnumber of total uses of the detected limited-use surgical tool has beenreached; and if so, generating a tool-expiration notification to becommunicated to surgical staff.
 14. An apparatus for automaticallytracking usages of robotic surgery tools, the apparatus comprising: oneor more processors; a memory coupled to the one or more processors, thememory storing instructions that, when executed by the one or moreprocessors, cause the apparatus to: receive a surgical video capturedduring a robotic surgery; process the surgical video to detect alimited-use surgical tool in the surgical video; determine whether thedetected limited-use surgical tool has been engaged in the roboticsurgery, which includes determining if the detected limited-use surgicaltool continues to present in the surgical video for at least apredetermined minimum time period by tracking the detected limited-usesurgical tool in the surgical video for a predetermined number ofconsecutive video frames corresponding to the predetermined minimum timeperiod; and if so, determine if the detected limited-use surgical toolhas been previously-engaged in the robotic surgery: if not, increment atotal-engagement count of the detected limited-use surgical tool;otherwise, continue monitoring the detected limited-use surgical tool insurgical video.
 15. The apparatus of claim 14, wherein the memoryfurther stores instructions that, when executed by the one or moreprocessors, cause the apparatus to activate a tool-usage countingprocess to track and record a number of uses of the detected limited-usesurgical tool during the tool engagement after determining that thedetected limited-use surgical tool has been engaged.
 16. The apparatusof claim 14, wherein the memory further stores instructions that, whenexecuted by the one or more processors, cause the apparatus to use asecond machine-learning model to track the number of uses of thedetected limited-use surgical tool, wherein the second machine-learningmodel is trained to detect an event in the surgical video representing asingle use of the detected limited-use surgical tool.
 17. The apparatusof claim 14, wherein the memory further stores instructions that, whenexecuted by the one or more processors, cause the apparatus to: receivean existing number of total uses of the detected limited-use surgicaltool from previous engagements of the detected limited-use surgicaltool; and update the total uses of the detected limited-use surgicaltool by combining the recorded number of uses during the tool engagementwith the existing number of total uses from the previous engagements.18. The apparatus of claim 15, wherein the memory further storesinstructions that, when executed by the one or more processors, causethe apparatus to: determine if a maximum number of total uses of thedetected limited-use surgical tool has been reached; and if so, generatea tool-expiration notification to be communicated to surgical staff. 19.A robotic surgical system, comprising: one or more surgical tools eachcoupled to a respective robotic arm; an endoscope configured to capturesurgical videos; and a controller coupled to each robotic arm and theendoscope and configured to: receive a surgical video captured by theendoscope during a robotic surgery; process the surgical video to detecta limited-use surgical tool in the surgical video; determine whether thedetected limited-use surgical tool has been engaged in the roboticsurgery, which includes determining if the detected limited-use surgicaltool continues to present in the surgical video for at least apredetermined minimum time period by tracking the detected limited-usesurgical tool in the surgical video for a predetermined number ofconsecutive video frames corresponding to the predetermined minimum timeperiod; and if so, determine whether the detected limited-use surgicaltool has been previously engaged during the robotic surgery: if not,increment a total-engagement count of the detected limited-use surgicaltool; otherwise, continue monitoring the detected limited-use surgicaltool in the surgical video.
 20. The robotic surgical system of claim 19,wherein the controller is further configured to: detect that thetotal-engagement count has reached a maximum number of engagements forthe safe and/or effective use of the detected limited-use surgical tool;and mark the detected limited-use surgical tool as expired and update astatus of the detected surgical tool for tool inventory purposes.